[Kernel-spark]Refactor to introduce Snapshot Manager abstraction and remove table path reference #5395
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
This PR introduces architectural improvements to prepare for CCv2 (catalog managed table) support, no functional changes.
1. Introduce
SnapshotManagerAbstractionSnapshotManagerinterface to encapsulate snapshot operationsPathBasedSnapshotManager(current path-based implementation)StreamingHelpertoPathBasedSnapshotManagergetTableChanges()method to encapsulateCommitRangeBuilderlogicgetMetadata()method to avoidSnapshotImplcasting2. Remove Direct
tablePathDependenciesRemoved
tablePathreferences from:SparkScanBuilder: Now receivesSnapshotManagerSparkScan: Derives path fromScanStateRow.getTableRoot()when neededSparkBatch: Completely removedtablePathfieldSparkMicroBatchStream: UsesSnapshotManager.getTableChanges()This refactor allows future
SnapshotManagerimplementations for CCv2 without relying on file system paths.How was this patch tested?
StreamingHelperTest→PathBasedSnapshotManagerTestSparkScanBuilderTestandSparkMicroBatchStreamTestto use new architectureDoes this PR introduce any user-facing changes?
No